Method and System for Customizing Camera Parameters

ABSTRACT

An apparatus for customizing camera parameters is provided. The apparatus includes: an image signal processor; and a processor configured to execute instructions to: generate a first edited image based on a first original image, the first original image including first original image data and first metadata indicating a first pattern of processed RGB colors; identify a first mapping function by analyzing the first pattern with respect to a fixed pattern of unprocessed-RGB color space samples; generate first unprocessed image data based on the first mapping function and the first original image; identify image signal processor adjustment parameters based on the first unprocessed image data and the first edited image; and image signal processor parameters of the image signal processor based on the image signal processor adjustment parameters.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to U.S. Provisional Patent Application No. 62/993,647, filed Mar. 23,2020, in the U.S. Patent and Trademark Office, and U.S. ProvisionalPatent Application No. 63/046,550, filed Jun. 30, 2020 in the U.S.Patent and Trademark Office, the disclosures of which are incorporatedby reference herein in their entireties.

BACKGROUND 1. Field

The disclosure relates to customizing parameter settings of an imagesignal processor (ISP) through indirect observation of actions relatedto image editing and image preferences.

2. Description of Related Art

A mobile device may include a camera having an image sensor and an imagesignal processor (ISP). The image signal processor is dedicated hardwarethat renders a raw sensor image to a final output image encoded to adisplay-referred format. The unprocessed sensor image may be referred toas a raw-RGB image and the ISP-rendered display-referred image may bereferred to as a processed display-referred image. When a user of themobile device uses the camera to capture an image of a scene, the imagesensor may generate a raw-RGB (i.e., unprocessed) image of the scene.The image sensor may provide the raw-RGB image in a raw image format tothe image signal processor for processing.

Professional photographers often prefer to shoot in an unprocessedraw-RGB image format. Shooting in raw-RGB provides users the ability torender the raw-RGB image to user controlled styles via photo-editingsoftware. In addition, certain image processing tasks, such as imagedeblurring, white-balance manipulation, and photometric stereo are moreeffective when applied to images in a raw-RGB format. While raw-RGBoffers advantages over display-referred images, most consumers still donot shoot in a raw format. One reason is because raw images aretypically 4-6 times larger than display-referred encoded images.Moreover, raw-format images are not easily shared as they requirerendering to a display-referred color space before viewing. Due to thesedrawbacks, the vast major of captured images are saved in adisplay-referred format.

In this regard, the image signal processor may process the raw image byperforming a number of routines to the sensor's raw-RGB image. Theseroutines may include white-balance, chromatic adaptation, colormanipulation, demosaicing, noise reduction, image sharpening, and thelike. For example, the image signal processor may process the raw imagethrough an image signal processor pipeline. Further, the image signalprocessor may output a processed display-referred image in a format suchas Tagged Image File Format (TIFF), Joint Photographic Experts Group(JPEG), and the like. The display-referred image may be encodedaccording to an industry standard color space, such as sRGB, Adobe RGBor ProPhoto RGB. The processed display-referred image may be stored,transmitted, or processed further via post processing. The unprocessedimage data may be, for example, 10-bit data, 12-bit data, 14-bit data or16-bit data, and the processed display-referred image may be, forexample, 8-bit data, 10-bit data, 12-bit data, 14-bit data or 16-bitdata. The raw-RGB image data is typically discarded after beingprocessed, and is not saved on a device.

The image signal processor may be configured with parameter settings,and may process raw images from the image sensor in accordance with theparameter settings. For example, the routines performed by the ISP maybe performed in accordance with the parameter settings, and the visualappearance of the processed display-referred image may depend on theparticular parameter settings of the ISP. Accordingly, ISPs that areconfigured with different parameter settings may generate processeddisplay-referred images that include visually different appearances andthereby impart a particular aesthetic to a captured image.

Typically, an image signal processor in a device is preconfigured withparticular and fixed parameter settings. The fixed parameter settingsmay be set in the factory. The same image signal processor may beimplemented in devices provided by different manufacturers. Eachmanufacture may provide a unique set of the fixed parameter settings. Inthis regard, first devices that are manufactured by a first company andsecond devices that are manufactured by a second company may include thesame ISP. However, because the first devices and the second devices havedifferent fixed parameter settings for the ISP, display-referred imagesgenerated by the first devices have a different appearance thandisplay-referred images generated by the second devices. Additionally,some manufacturers have different fixed parameter settings which varybased on geolocation. For example, a mobile device that is manufacturedfor North America may have a first set of fixed parameter settings,whereas a mobile device that is manufactured by the same company forAsia may have a second set of fixed parameter settings. Each set offixed parameter settings is set by “golden eye” experts that identifythe parameter settings in order to obtain desired image characteristics.Accordingly, a user of a mobile device cannot adjust the fixed parametersettings of the ISP. In this way, the customizability, functionality andextensibility of the mobile device are inhibited, and the userexperience is reduced.

Also, an image signal processor may include a large number of parametersettings. Moreover, the manner in which the parameters affect imageprocessing may not be apparent to a user. Therefore, even if it werepossible to manually change the parameter settings, such a process wouldbe time-consuming, error-prone, and/or difficult for a user of themobile device.

SUMMARY

To address the foregoing technical problems, embodiments of the presentdisclosure relate to indirectly learning photo-finishing preferencesbased on how images are edited or based on characteristics of imagesthat are identified. The parameter settings of the image signalprocessor can be modified based on these preferences to customizeonboard processing of an unprocessed (or raw) image.

One or more embodiments provide a mobile device that identifiesparameter setting adjustments by indirectly observing how images areedited using editing software on the mobile device.

One or more embodiments provide a mobile device that identifiesparameter setting adjustments by indirectly observing the types ofimages that are identified on the mobile device. For example, an imagemay be identified when the image is “liked” or “favorited” on a socialmedia platform.

One or more embodiments provide a mobile device that includes a sensorwhich generates raw-RGB image data that is then processed by an imagesignal processor to produce a display-referred image. When adisplay-referred image is edited, an original display referred image maybe stored with an edited display referred image. The mobile device mayconvert the original display-referred image back to its raw format usinga plurality of approaches. The image signal processor parameters may beupdated in a manner that when applied to the original display-referredimage an image is generated that is visually similar to the after-editdisplay referred image. The image signal processor parameters may beupdated on the camera hardware such that future captured images will beprocessed in customized manner.

One or more embodiments provide a mobile device that is able to generateraw-RGB image data based on metadata indicating a pattern of processedcolors.

One or more embodiments provide a mobile device that is able to generateraw-RGB image data based on metadata indicating sampled raw-RGB pixelvalues corresponding to known spatial locations in the display-referredimage.

One or more embodiments provide a mobile device that is able to use atrained neural network to generate raw-RGB image data based on adisplay-referred image. The neural network may be trained using pairs ofraw-RGB images and corresponding processed display-referred images.

According to embodiments of the disclosure, an apparatus for customizingcamera parameters includes: an image sensor; an image signal processor;a touch display; a memory storing instructions; and a processorconfigured to execute the instructions to: provide, via the touchdisplay, an image editing interface; generate, via the image editinginterface, a first edited image based on a first original image, whereinthe first original image comprises first original image data and firstmetadata indicating a first pattern of processed RGB colors; identify afirst mapping function by analyzing the first pattern with respect to afixed pattern of unprocessed-RGB color space samples; generate firstunprocessed image data based on the first mapping function and the firstoriginal image; identify a first plurality of image signal processoradjustment parameters based on the first unprocessed image data and thefirst edited image; update a plurality of image signal processorparameters of the image signal processor based on the first plurality ofimage signal processor adjustment parameters; and control the imagesignal processor to generate, using the updated plurality of imagesignal processor parameters, a second original image comprising secondoriginal image data and second metadata indicating a second pattern ofprocessed RGB colors based on second unprocessed image data receivedfrom the image sensor.

The processor may be further configured to execute the instructions togenerate the second pattern of processed RGB colors by processing thefixed pattern using the updated plurality of image signal processorparameters together with the second unprocessed image data.

The processor may be further configured to execute the instructions to:provide, via the touch display, an image gallery interface comprising aplurality of gallery images; receive, via the image gallery interface,an indication corresponding to a first gallery image from among theplurality of gallery images; generate first gallery unprocessed imagedata corresponding to the first gallery image by analyzing the firstgallery image using a neural network; identify a second plurality ofimage signal processor adjustment parameters based on the first galleryunprocessed image data and the first gallery image indicated via theimage gallery interface; and update the plurality of image signalprocessor parameters of the image signal processor based on the secondplurality of image signal processor adjustment parameters.

The processor may be further configured to execute the instructions to:control the image signal processor to sample third unprocessed imagedata received from the image sensor; control the image signal processorto generate third original image data by processing the thirdunprocessed image data using the updated plurality of image signalprocessor parameters; and generate a third image by appending thesamples of the third unprocessed image data to the third original imagedata.

The unprocessed-RGB color space samples of the fixed pattern representuniform samples of a full color space of the image sensor, eachprocessed RGB color of the first pattern may respectively correspond toone unprocessed-RGB color space sample of the fixed pattern, and theprocessor may be further configured to identify the first mappingfunction by comparing each processed RGB color of the first pattern witha corresponding unprocessed-RGB color space sample of the fixed pattern.

The processor may be further configured to execute the instructions toidentify the first plurality of image signal processor adjustmentparameters using a neural network, and the neural network may be trainedbased on a plurality of unprocessed images, each of which is processedusing a plurality of image signal processor parameters.

The plurality of image signal processor parameters may include a firstplurality of image signal processor parameters corresponding to a firstimage category and a second plurality of image signal processorparameters corresponding to a second image category, and the processoris further configured to execute the instructions to: identify whetherthe first edited image corresponds to the first image category or thesecond image category; update the first plurality of image signalprocessor parameters according to the first plurality of image signalprocessor adjustment parameters based on the first edited imagecorresponding to the first image category; and update the secondplurality of image signal processor parameters according to the firstplurality of image signal processor adjustment parameters based on thefirst edited image corresponding to the second image category.

The processor may be further configured to execute the instructions toindividually identify each of the first plurality of image signalprocessor adjustment parameters.

A plurality of pre-defined sets of parameters may be stored in thememory. Each of the plurality of pre-defined sets of parameters mayinclude a plurality of parameters respectively corresponding to theplurality of image signal processor parameters. The processor may befurther configured to execute the instructions to: identify one of theplurality of pre-defined sets of parameters based on the firstunprocessed image data and the first edited image; and identify thefirst plurality of image signal processor adjustment parameters based onthe plurality of parameters of the identified pre-defined set ofparameters.

According to embodiments of the disclosure a method of customizingparameters of an image signal processor of a mobile device includes:providing, via a touch display, an image editing interface; generating,via the image editing interface, a first edited image based on a firstoriginal image, wherein the first original image comprises firstoriginal image data and first metadata indicating a first pattern ofprocessed RGB colors; identifying a first mapping function by analyzingthe first pattern with respect to a fixed pattern of unprocessed-RGBcolor space samples; generating first unprocessed image data based onthe first mapping function and the first original image; identifying afirst plurality of image signal processor adjustment parameters based onthe first unprocessed image data and the first edited image; updating aplurality of image signal processor parameters of the image signalprocessor based on the first plurality of image signal processoradjustment parameters; and generating, using the updated plurality ofimage signal processor parameters, a second original image comprisingsecond original image data and second metadata indicating a secondpattern of processed RGB colors based on second unprocessed image datareceived from an image sensor.

The method may further include generating the second pattern ofprocessed RGB colors by processing the fixed pattern using the updatedplurality of image signal processor parameters together with the secondunprocessed image data.

The method may further include: providing, via the touch display, animage gallery interface comprising a plurality of gallery images;receiving, via the image gallery interface, an indication correspondingto a first gallery image from among the plurality of gallery images;generating first gallery unprocessed image data corresponding to thefirst gallery image by analyzing the first gallery image using a neuralnetwork; identifying a second plurality of image signal processoradjustment parameters based on the first gallery unprocessed image dataand the first gallery image indicated via the image gallery interface;and updating the plurality of image signal processor parameters of theimage signal processor based on the second plurality of image signalprocessor adjustment parameters.

The method may further include: sampling third unprocessed image datareceived from the image sensor; generating third original image data byprocessing the third unprocessed image data using the updated pluralityof image signal processor parameters; and generating a third image byappending the samples of the third unprocessed image data to the thirdoriginal image data.

The unprocessed-RGB color space samples of the fixed pattern representuniform samples of a full color space of the image sensor, eachprocessed RGB color of the first pattern may respectively correspond toone unprocessed-RGB color space sample of the fixed pattern, and theidentifying the first mapping function may include comparing eachprocessed RGB color of the first pattern with a correspondingunprocessed-RGB color space sample of the fixed pattern.

The method may further include identifying the first plurality of imagesignal processor adjustment parameters using a neural network, and theneural network may be trained based on a plurality of unprocessedimages, each of which is processed using a plurality of image signalprocessor parameters.

The plurality of image signal processor parameters may include a firstplurality of image signal processor parameters corresponding to a firstimage category and a second plurality of image signal processorparameters corresponding to a second image category, and the method mayfurther include: identifying whether the first edited image correspondsto the first image category or the second image category; updating thefirst plurality of image signal processor parameters according to thefirst plurality of image signal processor adjustment parameters based onthe first edited image corresponding to the first image category; andupdating the second plurality of image signal processor parametersaccording to the first plurality of image signal processor adjustmentparameters based on the first edited image corresponding to the secondimage category.

The identifying the first plurality of image signal processor adjustmentparameters may include individually identifying each of the firstplurality of image signal processor adjustment parameters.

The identifying the first plurality of image signal processor adjustmentparameters may include: identifying one of a plurality of pre-definedsets of parameters based on the first unprocessed image data and thefirst edited image, wherein each of the plurality of pre-defined sets ofparameters comprises a plurality of parameters respectivelycorresponding to the plurality of image signal processor parameters; andidentifying the first plurality of image signal processor adjustmentparameters based on the plurality of parameters of the identifiedpre-defined set of parameters.

According to embodiments of the disclosure a non-transitorycomputer-readable storage medium stores instructions configured to causea processor to: provide, via a touch display, an image editinginterface; generate, via the image editing interface, a first editedimage based on a first original image, wherein the first original imagecomprises first original image data and first metadata indicating afirst pattern of processed RGB colors; identify a first mapping functionby analyzing the first pattern with respect to a fixed pattern ofunprocessed-RGB color space samples; generate first unprocessed imagedata based on the first mapping function and the first original image;identify a first plurality of image signal processor adjustmentparameters based on the first unprocessed image data and the firstedited image; update a plurality of image signal processor parameters ofan image signal processor based on the first plurality of image signalprocessor adjustment parameters; and generate, using the updatedplurality of image signal processor parameters, a second original imagecomprising second original image data and second metadata indicating asecond pattern of processed RGB colors based on second unprocessed imagedata received from an image sensor.

The instructions may be further configured to cause the processor togenerate the second pattern of processed RGB colors by processing thefixed pattern using the updated plurality of image signal processorparameters together with the second unprocessed image data.

The instructions may be further configured to cause the processor to:provide, via the touch display, an image gallery interface comprising aplurality of gallery images; receive, via the image gallery interface,an indication corresponding to a first gallery image from among theplurality of gallery images; generate first gallery unprocessed imagedata corresponding to the first gallery image by analyzing the firstgallery image using a neural network; identify a second plurality ofimage signal processor adjustment parameters based on the first galleryunprocessed image data and the first gallery image indicated via theimage gallery interface; and update the plurality of image signalprocessor parameters of the image signal processor based on the secondplurality of image signal processor adjustment parameters.

The instructions may be further configured to cause the processor to:sample third unprocessed image data received from the image sensor;generate third original image data by processing the third unprocessedimage data using the updated plurality of image signal processorparameters; and generate a third image by appending the samples of thethird unprocessed image data to the third original image data.

The unprocessed-RGB color space samples of the fixed pattern representuniform samples of a full color space of the image sensor, eachprocessed RGB color of the first pattern may respectively correspond toone unprocessed-RGB color space sample of the fixed pattern, and theinstructions may be further configured to cause the processor toidentify the first mapping function by comparing each processed RGBcolor of the first pattern with a corresponding unprocessed-RGB colorspace sample of the fixed pattern.

The instructions may be further configured to cause the processor toidentify the first plurality of image signal processor adjustmentparameters using a neural network, and the neural network may be trainedbased on a plurality of unprocessed images, each of which is processedusing a plurality of image signal processor parameters.

The instructions may be further configured to cause the processor toidentify the first plurality of image signal processor adjustmentparameters by individually identifying each of the first plurality ofimage signal processor adjustment parameters.

The instructions may be further configured to cause the processor toidentify the first plurality of image signal processor adjustmentparameters by: identifying one of a plurality of pre-defined sets ofparameters based on the first unprocessed image data and the firstedited image, wherein each of the plurality of pre-defined sets ofparameters comprises a plurality of parameters respectivelycorresponding to the plurality of image signal processor parameters; andidentifying the first plurality of image signal processor adjustmentparameters based on the plurality of parameters of the identifiedpre-defined set of parameters.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and aspects of embodiments of thedisclosure will be more apparent from the following description taken inconjunction with the accompanying drawings, in which:

FIG. 1 is a diagram of an overview of a process or customizing parametersettings of an image signal processor of a camera of a mobile devicebased on a user edited image according to embodiments.

FIG. 2 is a diagram of components of a mobile device according toembodiments.

FIG. 3 is a diagram of an overview of a process for editing an imageaccording to embodiments.

FIGS. 4A, 4B and 4C show examples of generating image data with themetadata according to embodiments.

FIGS. 5A, 5B, 5C and 5D show examples of generating, or recovering,unprocessed image data based on processed display-referred image dataaccording to embodiments.

FIGS. 6A and 6B show examples of identifying image signal processoradjustment parameters according to embodiments.

FIG. 7 shows an example of updating image process processor parametersaccording to embodiments.

FIG. 8 is a diagram of an overview of a process or customizing parametersettings of an image signal processor of a camera of a mobile device byindirectly observing the types of images that are identified on themobile device according to embodiments.

FIG. 9 illustrates an example of an image being indicated according toembodiments.

FIG. 10 shows an example of generating, or recovering, unprocessed imagedata corresponding to an indicated image according to embodiments.

FIG. 11 illustrates an example of generating parameters for differentimage categories according to embodiments.

FIG. 12 illustrates an example of updating image signal processorparameters for each of the P image categories according to embodiments.

FIG. 13 is a flowchart of a method 2000 of customizing image signalprocessor parameters according to embodiments.

DETAILED DESCRIPTION

The following detailed description of example embodiments refers to theaccompanying drawings. The same reference numbers in different drawingsmay identify the same or similar elements.

Expressions such as “at least one of,” when preceding a list ofelements, modify the entire list of elements and do not modify theindividual elements of the list. For example, the expression, “at leastone of a, b, and c,” should be understood as including only a, only b,only c, both a and b, both a and c, both b and c, or all of a, b, and c.

FIG. 1 is a diagram of an overview of a process or customizing parametersettings of an image signal processor of a camera of a mobile devicebased on a user edited image according to embodiments.

As shown in FIG. 1, an original image, which may be a processeddisplay-referred image that is encoded in a display-referred formataccording to an industry standard color space, such as sRGB, Adobe RGBor ProPhoto RGB, may be edited in a process 110. The process 110 may beperformed on a mobile device. For example, the original image may beedited through a user interface for editing images that is provided onthe mobile device. Based on interactions with the user interface,various characteristics of the original image may be modified. Forexample, the process 110 may modify brightness, exposure, contrast,saturation, hue or other characteristics of the original image.

When process 110 has been completed, an edited version of the originalimage is generated and stored in association with the original image.The edited version of the original image may be encoded in adisplay-referred format. For example, the original image and the editedversion of the original image may form an image pair.

Process 120 is performed to generate unprocessed image data based on theoriginal image. For example, the process 120 may be initiated when theprocess 110 is completed. The unprocessed image data may be similar tounprocessed, or raw, image data that is generated by an image sensor ofthe camera of the mobile device. In this regard, process 120 isperformed to undo processing performed by an ISP to recover the rawimage data generated by an image sensor. The unprocessed image datagenerated in process 120 may be recovered raw image data that is thesame as or substantially similar to the raw image data generated by theimage sensor. Various methods for generating the unprocessed image datawill be discussed below in detail.

Process 130 is performed to identify image signal processor adjustmentparameters based on the edited image generated in process 110 and theunprocessed image data obtained in the process 120. The identified imagesignal processor adjustment parameters may be used to adjust theinternal parameter settings of the image signal processor so that theimage signal processor outputs images more similar to the edited image.

For example, the parameters of the image signal processor may includewhite-balance, chromatic adaptation, color manipulation, de-mosaicing,noise reduction, image sharpening, chroma-enhancement, skin-enhancement,select-color enhancement. Embodiments are not limited to these specificparameters, and additional adjustment parameters may be identified forother parameters of the image signal processor. For example, the process130 may identify one or more of the parameters of the image signalprocessor that may be changed in order for the image signal processor togenerate images more similar to the edited image. The one or moreparameters may be identified based on the edited image and the recoveredraw image data. For each of the identified parameters, an adjustmentparameter may be identified. For example, each of the image signalprocessor adjustment parameters may indicate a change to be made to acorresponding image signal processor parameter.

Process 140 is performed to change, or update, one or more of the imagesingle processor parameters currently set in the image signal processoraccording to the image signal processor adjustment parameters. Once theimage signal processor parameters have been updated, the image signalprocessor has been customized.

Subsequently, when the camera of the mobile device is used to captureimages, the image signal processor will process raw image signalsreceived from the image sensor according to the updated image signalprocessor parameters. In this regard, a user is able to customize theonboard photo-finishing process performed by the image signal processorand thereby customize the aesthetic of images processed by the imagesignal processor.

FIG. 2 is a diagram of components of a mobile device according toembodiments. As shown in FIG. 2, the mobile device 1000 may include aprocessor 1100, a camera 1200 including an image signal processor 1210which operates based on parameters 1220 and an image sensor 1240, amemory 1300, a user interface (UI) 1400, a display 1500, and acommunication interface 1600.

The processor 1100 may be configured to adjust the parameters 1220 ofthe image signal processor 1210 of the camera 1200 of the mobile device1000. For example, the processor 1100 may control operation of theprocesses 110, 120, 130 and 140 discussed above. For example, theprocessor 1100 may be a central processing unit (CPU), a graphicsprocessing unit (GPU), an accelerated processing unit (APU), amicroprocessor, a microcontroller, a digital signal processor (DSP), afield-programmable gate array (FPGA), an application-specific integratedcircuit (ASIC), and the like.

The camera 1200 may be configured to capture an image of a scene. Thecamera 1200 may include the image signal processor 1210 and the imagesensor 1240. The image signal processor (ISP) 1210 may be configured toprocess an unprocessed (e.g., raw) image received from the image sensor1240 using the parameters 1220. For example, the image signal processor1210 may be, an image processing engine, an image processing unit (IPU),an integrated signal processor, a processor, and the like. The imagesensor 1240 may be configured to generate unprocessed (e.g., raw) imagedata based on light that is incident on the image sensor 1240. Forexample, the image sensor 1240 may be a charge coupled device (CCD) or acomplementary metal-oxide semiconductor (CMOS) sensor, and the like.

The parameters 1220 of the image signal processor 1210 may be settingsof parameters of the image signal processor 1210 that control how theimage signal processor 1210 processes unprocessed, or raw, image data.For example, the image signal processor 1210 may be configured withvarious parameters that may be associated with various settings.Accordingly, a particular combination of parameter settings mayconstitute a profile, and different permutations of parameter settingsmay constitute different profiles. The parameters may include, asexamples, white-balance, chromatic adaptation, color manipulation,chroma-enhancement, skin-enhancement, selective-color-enhancement, noisereduction, lens shading correction, color correction, gamma correction,sharpness enhancement, auto exposure correction, auto focus correction,de-mosaicing, color space conversion, hue, saturation, contrast, and thelike.

The memory 1300 may be configured to store instructions that, whenexecuted by the processor 1100, cause the processor 1100 to adjust theparameters of the image signal processor 1210 of the camera 1200 of themobile device 1000 based on a user input received via the UI 1400. Forexample, the memory 1300 may include a non-transitory computer-readablemedium such as a flash memory type memory, a hard disk type memory, amultimedia card micro type memory, a card type memory (e.g., securedigital (SD) or extreme digital (XD) memory), a random access memory(RAM), a static random access memory (SRAM), read-only memory (ROM),electrically erasable programmable read-only memory (EEPROM),programmable read-only memory (PROM), a magnetic memory, a magneticdisk, an optical disc, and the like.

The user interface (UI) 1400 may be configured to receive a userselection in relation to an image displayed via the display 1500. Forexample, the UI 1400 may be a touchscreen, a button, a dome switch, akeyboard, a keypad, a mouse, a switch, a microphone, a transceiver, asensor, and the like.

The display 1500 may be configured to display a plurality of images. Forexample, the display 1500 may be a liquid crystal display (LCD), anorganic light-emitting diode (OLED) display, and the like.

The communication interface 1600 may be configured to communicate withexternal devices. For example, the communication interface 1600 may be acellular interface, a Bluetooth interface, a wireless fidelity (Wi-Fi)interface, a Wi-Fi Direct (WFD) interface, a Bluetooth Low Energy (BLE)interface, an Infrared Data Association (IrDA) interface, a Near FieldCommunication (NFC) interface, a laser communication network interface,a universal serial bus (USB) interface, and Ethernet interface, anoptical interface, and the like.

The number and arrangement of components shown in FIG. 2 are provided asan example. In practice, the mobile device 1000 may include additionalcomponents, fewer components, different components, or differentlyarranged components than those shown in FIG. 2. A set of components(e.g., one or more components) of the mobile device 1000 may perform oneor more functions described as being performed by another set ofcomponents of the mobile device 1000.

FIG. 3 is a diagram of an overview of a process for editing an imageaccording to embodiments.

As shown in FIG. 3, original image 205 may be edited through editingprocess 210. The editing process 210 may be performed via a userinterface provided on a mobile device. As shown, a characteristic to bemodified, such as brightness, exposure, contrast, saturation, hue, orother characteristic, may be selected. The selected characteristic maybe modified by moving a sliding user interface element. The displayedimage may be updated as the sliding user interface element is moved inorder to provide visual feedback to a user. Once the image is edited,the edited image may be saved. For example, the edited image may besaved in association with the original image as an image pair.

When the process 110 is completed, process 120 may be performed togenerate unprocessed data based on the original image corresponding tothe edited image.

As discussed above, different methods may be used for generating theunprocessed image data in the process 120. Reverting a processeddisplay-referred image, such as a display-referred image encodedaccording to the sRGB industry standard, back to its originalunprocessed raw sensor state is a challenging problem. This is becausemany of the operations applied by the image signal processor arenonlinear and proprietary. The image signal processor is essentially a“blackbox” to the end user.

According to embodiments, the processed display-referred image mayinclude both data that is indicative of the original image, as well asmetadata indicative of processing performed by the image signalprocessor. The metadata may be subsequently used to generate, orrecover, the unprocessed image data.

The metadata may be generated by the image signal processor when theimage is captured. For example, the image signal processor may append afixed pattern of unprocessed RGB colors (i.e., raw-RGB colors) tounprocessed image data (i.e., an raw-RGB image) that is generated by theimage sensor. The fixed pattern may be independent of the image datagenerated by the image sensor.

The fixed pattern is then processed by the image signal processor alongwith the unprocessed image data generated by the image sensor. Theprocessed pattern corresponding to the fixed pattern is stored asmetadata with the processed display-referred image data.

The metadata may be used by the mobile device to generate unprocessedimage data because the metadata is representative of the processingperformed by the image signal processor. That is, because the pattern ofunprocessed RGB colors is a fixed pattern, a relationship between theknown fixed pattern and the metadata may be indicative of a relationshipbetween the processed display-referred image data and the unprocessed(i.e., raw) image data. Based on the relationship between the metadataand the fixed pattern, an inversion (or de-rendering) function may beidentified. In particular, the relationship may indicate how thespecific colors of the fixed pattern were modified by the image signalprocessor. The inversion function may be applied to the processeddisplay-referred image data to generate, or recover, unprocessed imagedata that is similar to the unprocessed image data generated by theimage sensor.

FIGS. 4A, 4B and 4C show examples of generating image data with themetadata according to embodiments.

For example, a fixed pattern 301 of unprocessed RGB colors may begenerated by uniformly sampling unprocessed-RGB color space. The fixedpattern 301 may be appended, at the time of capture, as a pixel mask forall captured images. This pattern may be rendered by the image signalprocessor along with the captured image.

For example, as shown in FIG. 4A, the fixed pattern 301 of unprocessedRGB colors (i.e., raw-RGB colors) may be appended to unprocessed imagedata 303 (i.e., a raw-RGB image) corresponding to a first image. Theimage signal processor may generate first image data 314 by processingby the fixed pattern 301 and the unprocessed image data 303corresponding to the first image. The image data output by the imagesignal processor may be a processed display-referred image and includeprocessed pattern 312, which corresponds to the fixed pattern 301, aswell as first processed display-referred image data 313 to form firstimage data 314. The display-referred image data corresponding to theprocessed pattern 312 may be stored as metadata of the first image data314.

For example, as shown in FIG. 4B, the same fixed pattern 301 ofunprocessed RGB colors may be appended to unprocessed image data 305corresponding to a second image. The image signal processor may generatesecond image 326 by processing by the fixed pattern 301 and theunprocessed image data 305 corresponding to the second image. Processedpattern 322 corresponding to the fixed pattern 301 may be appended tothe second processed display-referred image data 325 to form secondimage 326 The RGB image data corresponding to the processed pattern 322may be stored as metadata of the second image 326.

As shown, the fixed pattern 301 is the same for both the first image inFIG. 4A and the second image of FIG. 4B. The image signal processing310, performed by the image signal processor, processes the fixedpattern 301 and the unprocessed image data 303 according to firstsettings. The image signal processing 320, performed by the image signalprocessor, processes the fixed pattern 301 and the unprocessed imagedata 305 according to second settings. The first settings and the secondsettings are different. In this regard, the processed pattern 312appended to the first processed display-referred image data 313 is notthe same as the processed pattern 322 appended to the second processeddisplay-referred image data 325.

For example, the raw samples of the fixed pattern 301 may be selected byuniformly sampling the full space of unprocessed-RGB colors that theimage sensor is capable of producing. For example, an image sensor mayproduce raw-RGB image data with a bit depth of 10. In this case, thetotal number of unique raw-RGB color values that the image sensor canrecord is 1024×1024×1024. The 3D RGB volume is binned uniformly using32×32×32 samples. This fixed raw-RGB pattern may be appended to eachcaptured image and rendered through the image signal processor with thefull frame. The resulting 32×32×32 rendered RGB samples constitute themetadata. These additional pixels represent only 96 KB of overhead tothe full size image.

Embodiments are not limited to raw-RGB image data with a bit depth of10. In this regard, according to embodiments raw-RGB image data may be,for example, 12-bit raw-RGB image data, 14-bit raw-RGB image data, or16-bit raw-RGB image data. According to embodiments, the 3D RGB volumeof 12, 14 or 16 bit values may be binned uniformly using 32×32×32samples. According to embodiments, the 3D RGB volume may be binneduniformly using different samples, such as 16×16×16 samples, 64×64×64samples or 128×128×128 samples.

In the process 120, a mapping function, such as an inversion function orde-rendering function, may be identified based on a relationship betweenthe fixed pattern 301 and a processed pattern stored as metadata withprocessed display-referred image data. The mapping function may be usedto invert the processed display-referred image data to unprocessedraw-RGB image data. In particular, the relationship may indicate how theunprocessed image data was modified by the image signal processor. Theinversion function may be applied to the processed display-referredimage data to generate, or recover, unprocessed image data that issimilar to the unprocessed raw-RGB image data generated by the imagesensor.

As discussed above, embodiments relate to metadata that is generatedbased on the fixed pattern 301. However, embodiments are not limitedthereto. For example, according to embodiments, when an image iscaptured, processed display-referred image data may be generated by theimage signal processor based on the unprocessed image data received fromthe image sensor, and the unprocessed image data may be sampled andstored as metadata with processed display-referred image data. Forexample, the unprocessed image data may be sampled on a sparse grid andstored as the metadata.

The metadata corresponding to the unprocessed image data may be used bythe mobile device to generate unprocessed image data because arelationship between processed display-referred image data and theunprocessed image data stored as metadata is representative of theprocessing performed by the image signal processor.

Based on the relationship between the metadata and the processeddisplay-referred image data, a mapping function, such as an inversionfunction or de-rendering function, may be identified. In particular, therelationship may indicate how the specific samples of the unprocessedimage data were modified by the image signal processor. The inversionfunction may be applied to the processed display-referred image data togenerate, or recover, unprocessed image data that is similar to theunprocessed raw-RGB image data generated by the image sensor.

For example, as shown in FIG. 4C, unprocessed image data 307, which forexample may be 16-bit raw-RGB data, may be processed in image signalprocessing 330 to generate third processed display-referred image data317, which may be for example 8-bit RGB data. The unprocessed image data307 may also be sampled n times in image signal processing 340 togenerate unprocessed image samples 316. For example, the unprocessedimage samples 316 may include n samples, sample 316-1 through sample316-n, which may be sampled from the unprocessed image data 307 in theimage signal processing 340. Image signal processing 350 may beperformed to append the unprocessed image samples 316 to the thirdprocessed display-referred image data 317 as metadata to form thirdimage 336.

In the process 120, a mapping function may be identified based on theunprocessed image samples 316 and corresponding portions of the thirdprocessed display-referred image data 317. The mapping function may beused to invert the processed display-referred data to unprocessed (i.e.,raw-RGB) image data.

FIGS. 5A, 5B, 5C and 5D show examples of generating, or recovering,unprocessed image data based on processed display-referred image dataaccording to embodiments.

As discussed above, embodiments relate to generating unprocessed imagedata that is similar to raw-RGB data generated by the image sensor basedon processed display-referred image data and metadata appended to theprocessed display-referred image data.

As shown in FIG. 5A, the first image data 314 may include firstprocessed display-referred image data 313 and processed pattern 312. Theprocessed pattern 312 may correspond to the fixed pattern 301. Based onthe fixed pattern 301 and the processed pattern 312, a mapping function(i.e., an inversion or de-rendering function) may be identified based onthe unprocessed image data samples of the fixed pattern 301 and theircorresponding RGB values of the processed pattern 312. The mappingfunction may be used to generate, or recover, unprocessed image data 403in operation 410. For example, the unprocessed image data 403 maycorrespond to the unprocessed image data 303.

As shown in FIG. 5B, the second image 326 may include second processeddisplay-referred image data 325 and processed pattern 322. The processedpattern 322 may correspond to the fixed pattern 301. Based on the fixedpattern 301 and the processed pattern 322, a mapping function (i.e., aninversion or de-rendering function) may be identified based on theunprocessed image data samples of the fixed pattern 301 and theircorresponding RGB values of the processed pattern 322. The mappingfunction may be used to generate, or recover, unprocessed image data 405in operation 420. For example, the unprocessed image data 405 maycorrespond to the unprocessed image data 305.

For example, a reverse mapping function from processed display-referredimage data to raw-RGB may be identified based on rendered processeddisplay-referred image samples and their corresponding a priori knownfixed raw-RGB samples. The mapping may be implemented via a standardscatter point interpolation framework, and can be computed post-capturewhen the user edits the image.

Because the first image data 314 and the second image data 326 wereprocessed with different image signal processor settings, the reversemapping function in operation 410 will be different than the reversemapping function in operation 420.

As shown in FIG. 5C, the third image 336 may include third processeddisplay-referred image data 317 and unprocessed image samples 316. Theunprocessed image samples 316 may correspond to pixels of the thirdprocessed display-referred image data 317. Based on the unprocessedimage samples 316 and corresponding pixels of the third processeddisplay-referred image data 317, a mapping function (i.e., an inversionor de-rendering function) may be identified. The mapping function may beused to generate, or recover, unprocessed image data 407. For example,the unprocessed image data 407 may correspond to the unprocessed imagedata 307.

According to embodiments, unprocessed image data that is similar toraw-RGB data generated by the image sensor may be generated, orrecovered, from processed display-referred image data corresponding toan edited image. For example, a neural network may be used to generateunprocessed image data corresponding to an edited image that is notassociated with a corresponding original image.

As shown in FIG. 5D, a neural network 450 may be trained based on setsof display-referred image data 452 and corresponding raw image data 454.Each set may include display-referred image data and correspondingraw-RGB data. For example, display-referred image data 452-1 maycorrespond to raw-RGB data 454-1, and the display-referred image data452-1 and the raw-RGB data 454-1 may form a set. Similarly,display-referred image data 452-2 through 452-q may respectivelycorrespond to raw-RGB data 454-2 through 454-q. The neural network 450may process display-referred image data 456 and provide unprocessedimage data 458 that is similar to raw-RGB data generated by an imagesensor.

FIGS. 6A and 6B show examples of identifying image signal processoradjustment parameters according to embodiments.

As discussed above, process 130 may be performed to identify imagesignal processor adjustment parameters. The process 130 may be based onan original image that is a processed display-referred image and anedited version of the original image that is also a display-referredimage. Unprocessed image data that is similar to raw-RGB data generatedby the image sensor may be generated, or recovered, based on theoriginal image. The process 130 may analyze the unprocessed image dataand the edited image to identify how the image signal processor could bemodified to produce images that are more similar to the edited image.The parameters of the image signal processor may be different than editsto the original image.

For example, the image signal processor may have m adjustableparameters, m being a positive integer. The m parameters of the imagesignal processor may include white-balance, chroma-enhancement,skin-enhancement, selective-color-enhancement, noise reduction, lensshading correction, color correction, gamma correction, sharpnessenhancement, auto exposure correction, auto focus correction,de-mosaicing, color space conversion, hue, saturation, contrast, etc.Each of the parameters may have one or more settings that may be setwithin a bounded range.

For example, white-balance parameters of the image signal processor maycontrol whether a white color object in an image appears to be white. Inthis regard, the color of an object is affected by the lightingconditions under which the object is viewed. The white balanceparameters remove unrealistic color casts from an image. For example,the white balance parameters may include a color temperature, a redoffset, a red gain, a green offset, a green gain, a blue offset and ablue gain.

For example, chroma-enhancement parameters of the image signal processormay be control colorfulness or saturation of an image to be increased.For example, saturation parameters of the image signal processor maycontrol the saturation of different color components, such as red, greenand blue color components. For example, selective-color-enhancement ofthe image signal processor parameters may control intensity orsaturation of specific colors. For example, skin-enhancement parametersof the image signal processor may control enhance colors of human skin.

For example, hue parameters of the image signal processor may controlthe hue of red, green and blue color components.

For example, contrast parameters of the image signal processor maycontrol brightness levels between light and areas of an image.

For example, noise reduction parameters of the image signal processormay control how electronic noise, which increases with higher ISOsettings or slower shutter speeds, is removed from image signalinformation. For example, the noise reduction parameters may reducediverse color or pattern noise while preserving texture details.

For example, the lens correction parameters of the image signalprocessor may compensate for distortion due to the lens. The lenscorrection parameters may include lens shading correction parameters,chromatic aberration parameters, and lens distortion parameters.

For example, gamma correction parameters of the image signal processormay control how nonlinear information is generated based on linear imageinformation received form an image sensor.

For example, sharpness enhancement parameters of the image signalprocessor may control how the image signal processor adjusts sharpnessof an image. The sharpness enhancement parameters may include anintensity parameter, a radius parameter, and a detail parameter.

For example, auto exposure correction parameters of the image signalprocessor may control what shutter speeds, aperture settings and ISOsettings based on how much light is received at the image sensor.

For example, auto focus correction parameters of the image signalprocessor control how the image signal processor corrects adjustment offocus.

For example, de-mosaicing parameters of the image signal processor maycontrol how the image signal processor calculates and interpolates thecolor received by each pixel by analyzing color information around thepixel. For example, these parameters control how colors other than thepixel colors, which for example are limited to red, green and blue, areinterpreted.

For example, color space conversion parameters of the image signalprocessor may control how the image signal processor translates an imagefrom one color space to another color space.

As shown in FIG. 6A, an analysis process may be performed for each ofthe m parameters of the image signal processor. Embodiments are notlimited to performing analysis for each of the m parameters, andaccording to embodiments, analysis may be performed for any number orany combination of the m parameters. For example, analysis may beperformed for only one of the parameters, only two of the parameters,(m-1) of the parameters, or all of the parameters. Additionally,embodiments are not limited to the specific parameters discussed herein,and additional or modified parameters may be analyzed.

As shown in FIG. 6A, the process 130 includes analyzing each of theparameters P1 through Pm. The first parameter P1 is analyzed inoperation 131-1. In operation 131-1, it is identified whether the firstparameter P1 could be modified so that an image output by the imagesignal processor is more similar to the edited image. For example, thefirst parameter P1 may correspond to white balance. A parameter thatprovides an output most similar to the edited image may be identifiedfrom a white balance parameter space. A white balance adjustmentparameter may be identified based on the identified white balanceparameter and a current white balance parameter.

The second parameter P2 is analyzed in operation 131-2. In operation131-2, it is identified whether the second parameter P2 could bemodified so that an image output by the image signal processor is moresimilar to the edited image. For example, the second parameter P2 maycorrespond to saturation. A parameter that provides an output mostsimilar to the edited image may be identified from a saturationparameter space. A saturation adjustment parameter may be identifiedbased on the identified saturation parameter and a current saturationparameter.

The third parameter P3 is analyzed in operation 131-3. In operation131-3, it is identified whether the third parameter P3 could be modifiedso that an image output by the image signal processor is more similar tothe edited image. For example, the third parameter P3 may correspond tocontrast. A parameter that provides an output most similar to the editedimage may be identified from a contrast parameter space. A contrastadjustment parameter may be identified based on the identified contrastparameter and a current contrast parameter.

The mth parameter Pm is analyzed in operation 131-m. In operation 131-m,it is identified whether the mth parameter Pm could be modified so thatan image output by the image signal processor is more similar to theedited image. For example, the mth parameter Pm may correspond to skintone. A parameter that provides an output most similar to the editedimage may be identified from a skin tone parameter space. A skin toneadjustment parameter may be identified based on the identified skin toneparameter and a current skin tone parameter.

In operation 135, a mapping table corresponding to the results of eachanalysis may be generated. Image signal processor adjustment parametersmay be provided to the image signal processor. Embodiments are notlimited to a mapping table, and other processes for generating imagesignal processor adjustment parameters may be used. In some embodimentsthe current parameters of the image signal processor may be input to themapping table 135 or to the analysis operations 131.

Embodiments are not limited to analyzing each parameter individually.For example, according to embodiments a plurality of pre-defined sets ofparameters may be provided. As shown in FIG. 6B, analysis may beperformed on the plurality of pre-defined sets of parameters to identifya pre-defined set of parameters that would control the image signalprocessor to output an image similar to the edited image based on therecovered unprocessed image data. Although four sets of pre-defined setsof parameters are illustrated in FIG. 6B, embodiments are not limitedthereto. For example, there may be hundreds of pre-defined sets,thousands of pre-defined sets or tens of thousands of pre-defined sets.Each of the plurality of pre-defined sets may include a parameter foreach of the m parameters. For example, a first pre-defined set 132-1 mayinclude a first value for the first parameter P1, a first value for thesecond parameter P2, a first value for the third parameter P3 and afirst value for the mth parameter. A second pre-defined set 132-2 mayinclude a second value for the first parameter P1, a second value forthe second parameter P2, a second value for the third parameter P3 and asecond value for the mth parameter. A third pre-defined set 132-3 mayinclude a third value for the first parameter P1, a third value for thesecond parameter P2, a third value for the third parameter P3 and athird value for the mth parameter. A fourth pre-defined set 132-4 mayinclude a fourth value for the first parameter P1, a fourth value forthe second parameter P2, a fourth value for the third parameter P3 and afourth value for the mth parameter. Some of the parameters in differentsets may have the same value. For example, the fourth value for thefirst parameter P1 may be the same as the first value for the firstparameter P1.

According to embodiments, candidate images may be generated based oneach of the pre-defined sets of parameters. For example, first candidateimage 133-1 may be generated based on the first pre-defined set ofparameters 132-1 second candidate image 133-2 may be generated based onthe second pre-defined set of parameters 132-2, third candidate image133-3 may be generated based on the third pre-defined set of parameters132-3, and fourth candidate image 133-4 may be generated based on thefourth pre-defined set of parameters 132-4.

In operation 134, the candidate image that is most similar to the editedimage may be identified. Based on the identified candidate image, one ofthe pre-defined sets of parameters may be identified. The mapping table135 may be updated based on each of the parameters in the identifiedpre-defined set.

FIG. 7 shows an example of updating image process processor parametersaccording to embodiments. As shown, m adjustment parameters 141-1through 141-m are provided to the image signal processor in operation140. The image signal processor may then be updated based on thereceived image signal processor parameters.

Because the image signal processor has been updated based on theadjustment parameters, the image signal processor is now personalized.When a command to capture a new image is received, the image signalprocessor will operate based on the updated image signal processorparameters.

For example, if an image has been edited to appear brighter, then theprocess 130 may identify that the auto-exposure processing of the imagesignal processor could be adjusted so that a brighter image is output bythe image signal processor. Then, in the process 140 the image signalprocessor parameters are adjusted so that unprocessed image data isgenerally exposed to a higher level. As multiple images are edited overtime, similar edits to brightness levels may allow for the auto-exposureparameters of the image signal processor to be optimized according tothe user's preferences.

For example, if an image has been edited so that the colors appear moreyellow and/or red, then the process 130 may identify that the whitebalance parameters of the image signal processor could be adjusted sothat a warmer image is output by the image signal processor. Then, inthe process 140 the image signal processor parameters are adjusted sothat unprocessed image data is generally exposed to a warmer colortemperature. As multiple images are edited over time, similar edits toyellow and/or red colors may allow for the white balance parameters ofthe image signal processor to be optimized according to the user'spreferences.

According to embodiments, the adjustments to the image signal processorparameters may be made incrementally so that significant changes to theimage signal processor parameters are not made at one time. For example,each time a parameter is adjusted, the adjustment may be limited to athreshold percentage. For example, the threshold percentage may be 1%.By limiting the amount a parameter may be adjusted at one time, issuesarising when an image is accidentally modified in an unintended mannermay be avoided. For example, if an image is accidentally edited to goall “black”, the image signal processor parameters should not suddenlyprocess all future captured images to black. Embodiments are not limitedto the 1% threshold percentage, and other threshold percentages, such as5% or 10%, may be utilized. Additionally, according to embodiments thethreshold percentage may be changeable via a user interface.

According to embodiments, default image signal process parameters andcheck points of the image signal processor parameters may be stored onthe device so that the default parameters or previous parameters may bereverted to. For example, a check point may be added every 100 updates.Additionally, a check point may be manually added based on a userselection made via the user interface. In this regard, if the imagesignals process parameters are providing undesirable results, then theimage signals process parameters may be set to the default image signalsprocess parameters, or to those indicated by one of the check points.The image signals process parameters may be reverted to default imagesignals process parameters or those indicated by one of the check pointsby user input through the user interface.

FIG. 8 is a diagram of an overview of a process or customizing parametersettings of an image signal processor of a camera of a mobile device byindirectly observing the types of images that are identified on themobile device according to embodiments. For example, an image may beidentified when the image is “liked” or “favorited” on a social mediaplatform.

As shown, an image may be indicated in process 810. For example, animage may be indicated when the image is “liked” or “favorited” on asocial media platform. For example, the image may be indicated through auser interface. For example, the image may be a processeddisplay-referred image.

As shown, in process 820, unprocessed image data that is similar toraw-RGB data generated by the image sensor is generated when the imageis indicated. For example, if the image is indicated, process 820 may beperformed to generate unprocessed image data corresponding to theindicated image.

In process 830, image signal processor adjustment parameters may beidentified based on the indicated image and the unprocessed image datagenerated in process 820. For example, the process 830 may be performedusing an artificial intelligence-based image signal processor network.The artificial intelligence network may be used to estimate thenecessary image signal processor parameters that would make theunprocessed image data appear more like the indicated image. Thisartificial intelligence network may be learned by processing manyunprocessed (i.e., raw-RGB) sensor images with a wide range of imagesignal processor parameters.

FIG. 9 illustrates an example of an image being indicated in process810.

As shown in FIG. 9, images 801-1, 801-2, 801-3, 801-4 may be displayedon a user interface of the mobile device. One of the images may beindicated by user selection 803. For example, image 801-4 may beindicated by the user selection 803.

FIG. 10 shows an example of generating, or recovering, unprocessed imagedata corresponding to an indicated image according to embodiments.

As discussed above, embodiments relate to generating unprocessed imagedata based on processed display-referred image data and metadataappended to the processed display-referred image data. Embodiments alsorelate to generating unprocessed image data using a neural network.

Because the indicated image may not have been generated using the mobiledevice, the metadata discussed above for generating the unprocessedimage data may not be present. The mobile device may determine whetherthe metadata is present, and if so then the metadata may be used togenerate the unprocessed image data in a manner similar to the processesdiscussed above. If the metadata is not present, then an artificialintelligence-based neural network that has been trained with randomimages, such as those from the internet, to convert the indicated imageto unprocessed image data. Any metadata that is provided with theindicated image may also be used by the artificial intelligence network.

Processes 830 and process 840 may be substantially similar to process130 and process 140, respectively.

As different images are indicated through, for example, various socialmedia networks, the image signal processor of the mobile device becomescustomized and will produce images more similar to those indicated onthe social media networks.

According to embodiments, an image signal processor may be configured toprocess image data of different image categories based on differentimage signal processor parameters. For example, the image signalprocessor may include P sets of parameters for P image categories. P maybe a positive integer. The P image categories may include portraitimages, landscape images, food images, animal images, etc. Image signalprocessor parameters for each of the image categories may be stored onthe image processor. When an image is captured, the image may becategorized and processed according to image signal processor parameterscorresponding to one of the categories. According to embodiments, thecaptured image may be automatically categorized. According toembodiments, the image may be categorized based on a selection madethrough a user interface.

For example, when an image is edited in process 110, a category of theimage may be identified and the image signal processor parameters thatcorrespond to the identified category may be updated independent of theimage signal processor parameters that correspond to the othercategories.

For example, when an image is indicated in process 810, a category ofthe image may be identified and the image signal processor parametersthat correspond to the identified category may be updated independent ofthe image signal processor parameters that correspond to the othercategories.

FIG. 11 illustrates an example of generating parameters for differentimage categories according to embodiments.

As shown, process 136 may include P processes 136-1 through 136-P forgenerating image signal processor adjustment parameters for each of theP image categories. The category of an edited image (or an indicatedimage) may be identified in process 138. Based on which image categoryis identified, one of the P processes 136-1 through 136-P may beperformed. Image signal processor adjustment parameters for acorresponding image category may be identified by the one of the Pprocesses.

Although FIG. 11 illustrates performing an analysis process for each ofthe m parameters for each of the P categories, embodiments are notlimited thereto. For example, as discussed above with reference to FIG.6B, a plurality of pre-defined sets of parameters may be provided.Analysis may be performed on the plurality of pre-defined sets ofparameters to identify a pre-defined set of parameters that wouldcontrol the image signal processor to output an image similar to theedited image based on the recovered unprocessed image data. In asomewhat similar manner, according to embodiments, one of thepre-defined sets of parameters may be selected for each of the Pcategories.

FIG. 12 illustrates an example of updating image signal processorparameters for each of the P image categories according to embodiments.

As shown, process 146 may include P processes 146-1 through 146-P forupdating the image signal processor parameters of each of the P imagecategories. The adjustment parameters are identified for an individualimage category, and the image signal processor is updated in operation145 based on the adjustment parameters for a corresponding one of the Pimage categories.

FIG. 13 is a flowchart of a method 2000 of customizing image signalprocessor parameters according to embodiments.

The method 2000 may be performed by the mobile device 1000.

In operation 2005 the method includes providing, via a touch display, animage editing interface.

In operation 2010, the method includes generating, via the image editinginterface, a first edited image based on a first original image, whereinthe first original image comprises first original image data and firstmetadata indicating a first pattern of processed RGB colors.

In operation 2015, the method includes identifying a first mappingfunction by analyzing the first pattern with respect to a fixed patternof unprocessed-RGB color space samples.

In operation 2020, the method includes generating first unprocessedimage data based on the first mapping function and the first originalimage.

In operation 2025, the method includes identifying a first plurality ofimage signal processor adjustment parameters based on the firstunprocessed image data and the first edited image.

In operation 2030, the method includes updating a plurality of imagesignal processor parameters of the image signal processor based on thefirst plurality of image signal processor adjustment parameters.

In operation 2035, the method includes generating, using the updatedplurality of image signal processor parameters, a second original imagecomprising second original image data and second metadata indicating asecond pattern of processed RGB colors based on second unprocessed imagedata received from an image sensor.

The foregoing disclosure provides illustration and description, but isnot intended to be exhaustive or to limit the implementations to theprecise form disclosed. Modifications and variations are possible inlight of the above disclosure or may be acquired from practice of theimplementations.

As used herein, the term “component” is intended to be broadly construedas hardware, firmware, or a combination of hardware and software.

It will be apparent that systems and/or methods, described herein, maybe implemented in different forms of hardware, firmware, or acombination of hardware and software. The actual specialized controlhardware or software code used to implement these systems and/or methodsis not limiting of the implementations. Thus, the operation and behaviorof the systems and/or methods were described herein without reference tospecific software code—it being understood that software and hardwaremay be designed to implement the systems and/or methods based on thedescription herein.

Even though particular combinations of features are recited in theclaims and/or disclosed in the specification, these combinations are notintended to limit the disclosure of possible implementations. In fact,many of these features may be combined in ways not specifically recitedin the claims and/or disclosed in the specification. Although eachdependent claim listed below may directly depend on only one claim, thedisclosure of possible implementations includes each dependent claim incombination with every other claim in the claim set.

No element, act, or instruction used herein should be construed ascritical or essential unless explicitly described as such. Also, as usedherein, the articles “a” and “an” are intended to include one or moreitems, and may be used interchangeably with “one or more.” Furthermore,as used herein, the term “set” is intended to include one or more items(e.g., related items, unrelated items, a combination of related andunrelated items, etc.), and may be used interchangeably with “one ormore.” Where only one item is intended, the term “one” or similarlanguage is used. Also, as used herein, the terms “has,” “have,”“having,” or the like are intended to be open-ended terms. Further, thephrase “based on” is intended to mean “based, at least in part, on”unless explicitly stated otherwise.

While embodiments have been described with reference to the figures, itwill be understood by those of ordinary skill in the art that variouschanges in form and details may be made therein without departing fromthe spirit and scope as defined by the following claims.

What is claimed is:
 1. An apparatus for customizing camera parameters,the apparatus comprising: an image sensor; an image signal processor; atouch display; a memory storing instructions; and a processor configuredto execute the instructions to: provide, via the touch display, an imageediting interface; generate, via the image editing interface, a firstedited image based on a first original image, wherein the first originalimage comprises first original image data and first metadata indicatinga first pattern of processed RGB colors; identify a first mappingfunction by analyzing the first pattern with respect to a fixed patternof unprocessed-RGB color space samples; generate first unprocessed imagedata based on the first mapping function and the first original image;identify a first plurality of image signal processor adjustmentparameters based on the first unprocessed image data and the firstedited image; update a plurality of image signal processor parameters ofthe image signal processor based on the first plurality of image signalprocessor adjustment parameters; and control the image signal processorto generate, using the updated plurality of image signal processorparameters, a second original image comprising second original imagedata and second metadata indicating a second pattern of processed RGBcolors based on second unprocessed image data received from the imagesensor.
 2. The apparatus of claim 1, wherein the processor is furtherconfigured to execute the instructions to generate the second pattern ofprocessed RGB colors by processing the fixed pattern using the updatedplurality of image signal processor parameters together with the secondunprocessed image data.
 3. The apparatus of claim 1, wherein theprocessor is further configured to execute the instructions to: provide,via the touch display, an image gallery interface comprising a pluralityof gallery images; receive, via the image gallery interface, anindication corresponding to a first gallery image from among theplurality of gallery images; generate first gallery unprocessed imagedata corresponding to the first gallery image by analyzing the firstgallery image using a neural network; identify a second plurality ofimage signal processor adjustment parameters based on the first galleryunprocessed image data and the first gallery image indicated via theimage gallery interface; and update the plurality of image signalprocessor parameters of the image signal processor based on the secondplurality of image signal processor adjustment parameters.
 4. Theapparatus of claim 1, wherein the processor is further configured toexecute the instructions to: control the image signal processor tosample third unprocessed image data received from the image sensor;control the image signal processor to generate third original image databy processing the third unprocessed image data using the updatedplurality of image signal processor parameters; and generate a thirdimage by appending the samples of the third unprocessed image data tothe third original image data.
 5. The apparatus of claim 1, wherein theunprocessed-RGB color space samples of the fixed pattern representuniform samples of a full color space of the image sensor, wherein eachprocessed RGB color of the first pattern respectively corresponds to oneunprocessed-RGB color space sample of the fixed pattern, and wherein theprocessor is further configured to identify the first mapping functionby comparing each processed RGB color of the first pattern with acorresponding unprocessed-RGB color space sample of the fixed pattern.6. The apparatus according to claim 1, wherein the processor is furtherconfigured to execute the instructions to identify the first pluralityof image signal processor adjustment parameters using a neural network,and wherein the neural network is trained based on a plurality ofunprocessed images, each of which is processed using a plurality ofimage signal processor parameters.
 7. The apparatus according to claim1, wherein the plurality of image signal processor parameters comprise afirst plurality of image signal processor parameters corresponding to afirst image category and a second plurality of image signal processorparameters corresponding to a second image category, and wherein theprocessor is further configured to execute the instructions to: identifywhether the first edited image corresponds to the first image categoryor the second image category; update the first plurality of image signalprocessor parameters according to the first plurality of image signalprocessor adjustment parameters based on the first edited imagecorresponding to the first image category; and update the secondplurality of image signal processor parameters according to the firstplurality of image signal processor adjustment parameters based on thefirst edited image corresponding to the second image category.
 8. Theapparatus according to claim 1, wherein the processor is furtherconfigured to execute the instructions to individually identify each ofthe first plurality of image signal processor adjustment parameters. 9.The apparatus according to claim 1, wherein a plurality of pre-definedsets of parameters are stored in the memory, wherein each of theplurality of pre-defined sets of parameters comprises a plurality ofparameters respectively corresponding to the plurality of image signalprocessor parameters, and wherein the processor is further configured toexecute the instructions to: identify one of the plurality ofpre-defined sets of parameters based on the first unprocessed image dataand the first edited image; and identify the first plurality of imagesignal processor adjustment parameters based on the plurality ofparameters of the identified pre-defined set of parameters.
 10. A methodof customizing parameters of an image signal processor of a mobiledevice, the method comprising: providing, via a touch display, an imageediting interface; generating, via the image editing interface, a firstedited image based on a first original image, wherein the first originalimage comprises first original image data and first metadata indicatinga first pattern of processed RGB colors; identifying a first mappingfunction by analyzing the first pattern with respect to a fixed patternof unprocessed-RGB color space samples; generating first unprocessedimage data based on the first mapping function and the first originalimage; identifying a first plurality of image signal processoradjustment parameters based on the first unprocessed image data and thefirst edited image; updating a plurality of image signal processorparameters of the image signal processor based on the first plurality ofimage signal processor adjustment parameters; and generating, using theupdated plurality of image signal processor parameters, a secondoriginal image comprising second original image data and second metadataindicating a second pattern of processed RGB colors based on secondunprocessed image data received from an image sensor.
 11. The method ofclaim 10, further comprising generating the second pattern of processedRGB colors by processing the fixed pattern using the updated pluralityof image signal processor parameters together with the secondunprocessed image data.
 12. The method of claim 10, further comprising:providing, via the touch display, an image gallery interface comprisinga plurality of gallery images; receiving, via the image galleryinterface, an indication corresponding to a first gallery image fromamong the plurality of gallery images; generating first galleryunprocessed image data corresponding to the first gallery image byanalyzing the first gallery image using a neural network; identifying asecond plurality of image signal processor adjustment parameters basedon the first gallery unprocessed image data and the first gallery imageindicated via the image gallery interface; and updating the plurality ofimage signal processor parameters of the image signal processor based onthe second plurality of image signal processor adjustment parameters.13. The method of claim 10, further comprising: sampling thirdunprocessed image data received from the image sensor; generating thirdoriginal image data by processing the third unprocessed image data usingthe updated plurality of image signal processor parameters; andgenerating a third image by appending the samples of the thirdunprocessed image data to the third original image data.
 14. The methodof claim 10, wherein the unprocessed-RGB color space samples of thefixed pattern represent uniform samples of a full color space of theimage sensor, wherein each processed RGB color of the first patternrespectively corresponds to one unprocessed-RGB color space sample ofthe fixed pattern, and wherein the identifying the first mappingfunction comprises comparing each processed RGB color of the firstpattern with a corresponding unprocessed-RGB color space sample of thefixed pattern.
 15. The method according to claim 10, further comprisingidentifying the first plurality of image signal processor adjustmentparameters using a neural network, wherein the neural network is trainedbased on a plurality of unprocessed images, each of which is processedusing a plurality of image signal processor parameters.
 16. The methodaccording to claim 10, wherein the plurality of image signal processorparameters comprise a first plurality of image signal processorparameters corresponding to a first image category and a secondplurality of image signal processor parameters corresponding to a secondimage category, and wherein the method further comprises: identifyingwhether the first edited image corresponds to the first image categoryor the second image category; updating the first plurality of imagesignal processor parameters according to the first plurality of imagesignal processor adjustment parameters based on the first edited imagecorresponding to the first image category; and updating the secondplurality of image signal processor parameters according to the firstplurality of image signal processor adjustment parameters based on thefirst edited image corresponding to the second image category.
 17. Themethod according to claim 10, wherein the identifying the firstplurality of image signal processor adjustment parameters comprisesindividually identifying each of the first plurality of image signalprocessor adjustment parameters.
 18. The method according to claim 10,wherein the identifying the first plurality of image signal processoradjustment parameters comprises: identifying one of a plurality ofpre-defined sets of parameters based on the first unprocessed image dataand the first edited image, wherein each of the plurality of pre-definedsets of parameters comprises a plurality of parameters respectivelycorresponding to the plurality of image signal processor parameters; andidentifying the first plurality of image signal processor adjustmentparameters based on the plurality of parameters of the identifiedpre-defined set of parameters.
 19. A non-transitory computer-readablestorage medium storing instructions configured to cause a processor to:provide, via a touch display, an image editing interface; generate, viathe image editing interface, a first edited image based on a firstoriginal image, wherein the first original image comprises firstoriginal image data and first metadata indicating a first pattern ofprocessed RGB colors; identify a first mapping function by analyzing thefirst pattern with respect to a fixed pattern of unprocessed-RGB colorspace samples; generate first unprocessed image data based on the firstmapping function and the first original image; identify a firstplurality of image signal processor adjustment parameters based on thefirst unprocessed image data and the first edited image; update aplurality of image signal processor parameters of an image signalprocessor based on the first plurality of image signal processoradjustment parameters; and generate, using the updated plurality ofimage signal processor parameters, a second original image comprisingsecond original image data and second metadata indicating a secondpattern of processed RGB colors based on second unprocessed image datareceived from an image sensor.
 20. The non-transitory computer-readablestorage medium of claim 19, wherein the instructions are furtherconfigured to cause the processor to generate the second pattern ofprocessed RGB colors by processing the fixed pattern using the updatedplurality of image signal processor parameters together with the secondunprocessed image data.
 21. The non-transitory computer-readable storagemedium of claim 19, wherein the instructions are further configured tocause the processor to: provide, via the touch display, an image galleryinterface comprising a plurality of gallery images; receive, via theimage gallery interface, an indication corresponding to a first galleryimage from among the plurality of gallery images; generate first galleryunprocessed image data corresponding to the first gallery image byanalyzing the first gallery image using a neural network; identify asecond plurality of image signal processor adjustment parameters basedon the first gallery unprocessed image data and the first gallery imageindicated via the image gallery interface; and update the plurality ofimage signal processor parameters of the image signal processor based onthe second plurality of image signal processor adjustment parameters.22. The non-transitory computer-readable storage medium of claim 19,wherein the instructions are further configured to cause the processorto: sample third unprocessed image data received from the image sensor;generate third original image data by processing the third unprocessedimage data using the updated plurality of image signal processorparameters; and generate a third image by appending the samples of thethird unprocessed image data to the third original image data.
 23. Thenon-transitory computer-readable storage medium of claim 19, wherein theunprocessed-RGB color space samples of the fixed pattern representuniform samples of a full color space of the image sensor, wherein eachprocessed RGB color of the first pattern respectively corresponds to oneunprocessed-RGB color space sample of the fixed pattern, and wherein theinstructions are further configured to cause the processor to identifythe first mapping function by comparing each processed RGB color of thefirst pattern with a corresponding unprocessed-RGB color space sample ofthe fixed pattern.
 24. The non-transitory computer-readable storagemedium of claim 19, wherein the instructions are further configured tocause the processor to identify the first plurality of image signalprocessor adjustment parameters using a neural network, and wherein theneural network is trained based on a plurality of unprocessed images,each of which is processed using a plurality of image signal processorparameters.
 25. The non-transitory computer-readable storage medium ofclaim 19, wherein the instructions are further configured to cause theprocessor to identify the first plurality of image signal processoradjustment parameters by individually identifying each of the firstplurality of image signal processor adjustment parameters.
 26. Thenon-transitory computer-readable storage medium of claim 19, wherein theinstructions are further configured to cause the processor to identifythe first plurality of image signal processor adjustment parameters by:identifying one of a plurality of pre-defined sets of parameters basedon the first unprocessed image data and the first edited image, whereineach of the plurality of pre-defined sets of parameters comprises aplurality of parameters respectively corresponding to the plurality ofimage signal processor parameters; and identifying the first pluralityof image signal processor adjustment parameters based on the pluralityof parameters of the identified pre-defined set of parameters.